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Abstract 

In manifold learning, algorithms based on graph Laplacian constructed from data have received 
considerable attention both in practical applications and theoretical analysis. Much of the exist- 
ing work has been done under the assumption that the data is sampled from a manifold without 
boundaries and singularities or that the functions of interest are evaluated away from such points. 

At the same time, it can be argued that singularities and boundaries are an important aspect 
of the geometry of realistic data. Boundaries occur whenever the process generating data has 
a bounding constraint; while singularities appear when two different manifolds intersect or if a 
process undergoes a "phase transition", changing non-smoothly as a function of a parameter. 

In this paper we consider the behavior of graph Laplacians at points at or near boundaries and 
two main types of other singularities: intersections, where different manifolds come together and 
sharp "edges ", where a manifold sharply changes direction. We show that the behavior of graph 
Laplacian near these singularities is quite different from that in the interior of the manifolds. In 
fact, a phenomenon somewhat reminiscent of the Gibbs effect in the analysis of Fourier series, can 
be observed in the behavior of graph Laplacian near such points. Unlike in the interior of the do- 
main, where graph Laplacian converges to the Laplace-Beltrami operator, near singularities graph 
Laplacian tends to a first-order differential operator, which exhibits different scaling behavior as a 
function of the kernel width. One important implication is that while points near the singularities 
occupy only a small part of the total volume, the difference in scaling results in a disproportion- 
ately large contribution to the total behavior. Another significant finding is that while the scaling 
behavior of the operator is the same near different types of singularities, they are very distinct at a 
more refined level of analysis. 

We believe that a comprehensive understanding of these structures in addition to the standard 
case of a smooth manifold can take us a long way toward better methods for analysis of complex 
non-linear data and can lead to significant progress in algorithm design. 
Keywords: Graph Laplacian, singularities, limit analysis 



1. Introduction 

Dealing with high-dimensional non-linear data is one of the key challenges in modem data analy- 
sis. In recent years a class of methods based on the mathematical notion of a manifold has become 
popular in machine learning, starting with the papers Roweis and Saul. (2000); Tenenbaum et al. 
(2000). The underlying intuition is that a process with a small number of parameters will gener- 
ate a low-dimensional surface in the potentially very high-dimensional space of features and that 
this situation is ubiquitous in real-world data. This idea is captured nicely by the technical notion 
of a smooth embedded Riemannian manifold, which provides the first realistic model for general 
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non-linear data. Still it does not reflect certain important aspects of real data, which can be mathe- 
matically understood as singularities and boundaries. 

The most basic and, arguably, the most important singularity in real data 
is an intersection, where two different manifolds come together in a region 
of space. This often happens in classification where two classes with pre- 
sumably different structure can give rise to similar objects (consider, e.g., the 
similarity of MNIST digits "7" and "1" on the right). Another important type 
of singularity is a (co-dimension one) ''edge", which is ubiquitous in com- 
puter graphics (think of an edge of the surface of a table) and happens whenever the behavior of an 
underlying process changes rapidly beyond a certain point (a "phase transition"). Finally, bound- 
aries occur when there are bounding constraints on the underlying process (think how poses of the 
human body are constrained by the range of motion of the joints) or on the representation in the 
ambient space (e.g., pixels cannot have negative intensity). 

In this paper we provide an analysis of these three cases for Laplacian-based learning algo- 
rithms. It turns out that all of these singularities result in a behavior near the singularity markedly 
different from that inside the manifold, a phenomenon somewhat reminiscent of the Gibbs effect in 
Fourier analysis. In particular, the scaling of the operator is different near singularities. As a result 
of this scaling behavior, singularities cannot be ignored, even globally, despite the fact that rela- 
tively few points are located near them, since each of these points contributes disproportionately to 
the total operator. We also find that at a finer level of analysis these singularities have quite different 
effects, which ai^e discussed below. 

We believe that a comprehensive understanding of these structures in addition to the standard 
case of a smooth manifold can take us a long way toward better methods for analysis of complex 
non-linear data and can lead to significant progress in algorithm design. 

Related work. Methods based on graph Laplacians constructed from data have gained accep- 
tance for a range of inference tasks including clustering von Luxburg (2007), semi-supervised 
learning (e.g. Chapelle et al. (2006); Zhu (2006)) and dimensionality reduction Belkin and Niyogi 
(2003), as well as others. An analysis in Belkin and Niyogi (2003) provided a mathematical frame- 
work for many of these methods by connecting graph Laplacian constructed from data to the 
Laplace-Beltrami operator of the underlying manifold based on the relation of the Laplacian and 
the heat equation. That analysis was extended and generalized in Lafon (2004); Hein (2005); 
Coifman and Lafon (2006); Belkin and Niyogi (2008a); Singer (2006); Gine and Koltchinskii (2006); 
Hein et al. (2007); Belkin and Niyogi (2008b) providing a detailed understanding of graph Lapla- 
cians obtained from manifold data. While boundary effects have been studied in non-parametric 
kernel smoothing (see, e.g., (Hardle, 1992, Chapter 4.4)), they have not been considered in the 
Laplacian-related literature, although we note that boundary behavior for the infinite data case can 
be derived from the Taylor series expansion in Lemma 9 of Coifman and Lafon (2006). To the 
best of our knowledge, other singularities are not considered in that literature. (In fact, the reason 
causing the different scaUng behavior for intersection and edge types of singularities is somewhat 
different from that for the boundary case.) 

In related developments, a considerable amount of recent work has been aimed at understanding 
the case of intersecting linear spaces using various techniques from algebraic geometry to spectral 
clustering (e.g., Vidal et al. (2005); Chen and Lerman (2009); Lerman and Zhang (2010)) both in 
terms of algorithms and theoretical guarantees. In our setting, this is a special case of a singular 
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manifold with intersection singularities. Another related work is Goldberg et al. (2009), where the 
authors analyze a model based on a mixture of manifolds in the context of semi-supervised learning. 

We also note that there has been much recent interest in reconstructing topological invariants of 
manifolds and other spaces (see, e.g, Chazal et al. (2009); Chazal and Oudot (2008); Niyogi et al. 
(2008, 2011)). The line of work most related to our results is on learning stratified spaces, where 
multiple submanifolds (strata) are "glued" nicely together Aanjaneya et al. (2011); Bendich et al. 
(2012); Haro et al. (2008); Bendich et al. (2007), which provides an even more general model for a 
singular space. 

Summary of results. Consider the (appropriately scaled, see Section 2) graph Laplacian L„ ^ con- 
structed from n data points, using the Gaussian kernel with bandwidth t. It can be shown (see the 
references above) that as t tends to zero and n tends to infinity at an appropriate rate, L„ t converges 
to the Laplace-Beltrami operator A (a second order differential operator) inside the manifold. In 
this paper, we show that for singular manifolds L„ t exhibits a very different behavior near the sin- 
gularity set. Specifically, for a sufficiently smooth function / and a small t, within distance \/t of 
the singularity set, Ln,tf is approximated by -^D, where D is some first order derivative operator 
that will be described explicitly. The difference in scaling becomes crucial when t is small as the 
■^D term becomes dominant. To simplify the discussion, we will first assume that the data is infi- 
nite, and thus L„ t/ can be replaced by Ltf. Finite sample bounds and rates will be discussed later. 
The types of singularities considered in the paper are as follows: 

Boundary. At a point x near manifold boundary, Lt can be approximated 
by ^0(-p=)5n, where i? is the distance from X to the boundary and n is ^ ' ^boundary-type 



the unit vector toward the nearest boundary point (outward normal at the ^ / 



ntersectton-type 

/ 

L , 



boundary). 3n denotes the directional derivative, while (^{z) is a scalar 
function of the form <\){^z) = Ce~^ . 
Intersection and edge. The situation is more complicated for other types edge-typ^^^^^^jiT^ 
of singularities, where two different manifolds Vl\ and 1^2 intersect or 

come together. Given a point xi G consider its projection onto VI2 and its nearest neighbor 
xq in the singularity. Similarly to the boundary case, let ni and n2, be the directions to xq from x\ 
and X2 respectively and let R\ and R2 be the corresponding distances. Ltf{xi) can be approximated 
by -^<pi(^)dnif{xo) + ■^4>2{^)dn2f{xo)- Importantly, the form of the scalar functions 0i, 02 
is different for different types of singularities: for intersection-type singularity, we have: (j)i{z) = 
Aize''-^^ , while for edge-type we have' : (j)i{z) = Aize~^^ + Bie~^^ . Here the coefficients Ai, 
Bi and C depend on the angle between the manifolds and can be written explicitly. 

Significant points and observations: 

1. Scaling. Within 0(\/t) distance of the singularity, Lt is dominated by a differential operator 
different from the Laplace-Beltrami operator. Away from the singularity, the higher- valued {l/^/i) 
term fades and Lt becomes an approximation to the Laplace-Beltrami operator A. 

2. Contributions of singular points. Let the dimension of the manifold be d. The volume of 
points within \/t of the singularity is approximately ^/t volrf_i(5), where vold_i(5) is the d — 
1 dimensional volume of the singular points^. Fix a function /. As t ^ 0, the integral Ltf 
over the points within ^/t of the singularity does not disappear and, in fact, will converge to a 



1. For compactness, we are simplifying (/>'s for edge singularities by removing some fiigher order terms (see Section 3.3). 

2. The set of singular points S is always d— 1 dimensional (co-dimension one), for the boundary and "edge" singularities 
(although not necessarily for the intersection). 
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constant, depending on the function and the d—1 dimensional volume of the singularity vol(i_i(5). 
Moreover, because of the scaling, the contribution of those near-singular points to the L2 norm will 
tend to infinity. We remark that, while different from a technical point of view, this phenomenon is 
reminiscent to the Gibbs effect in Fourier series, where the effects of discontinuity do not vanish as 
approximation becomes more precise. 



3. Shape near the boundary. While the 




(a) boundary 




(b) intersection (c) edge 



+ ze 



scaling is common for all singularities, the 
type of singularity is reflected in the shape 
of the function 0, which is very different for 
different types of singularities. Somewhat 
over-simplifying matters, one can think of 

2 2 

(j) = e^^ for the boundary case, cp = ze~^ for the intersection singularity and (p = e~ 
for the edge (see figures above). These differences lead to quite distinctive patterns, when the 
operator is applied to a fixed function. For example, the directional derivative terms disappear at 
the intersection-type singular points. In fact for these points Lt converges to the usual Laplace- 
Beltrami operator. However, at points within \/t distance of these points we expect to see Ltf take 
both positive and negative values of magnitude \/t. We think that this difference may be a key to 
future algorithms. 

4. Rates and finite sample bounds. Convergence rates for t as well as finite sample bounds are 
provided in Section 4. Interestingly, and perhaps counter-intuitively, the scaling implies that fewer 
data points are required to ensure that „ is an accurate (relative) estimate of Lt near the boundary 
than inside the domain. 

5. Impact on eigenfunctions. Eigenfunctions of the graph Laplacian associated to data are used in a 
large number of applications. While the question of convergence is very subtle (see Belkin and Niyogi 
(2008b) for the proof in the case of a smooth manifold), our results for fixed functions provide an 
indication of the expected answer assuming the convergence holds. We provide a brief discussion 
of it in Section 5. It turns out that the boundary case automatically leads to Neumann boundary 
conditions (even though the operator does not "see" the boundary explicitly). Quite surprisingly, 
the edge singularity appears to have no effect on the limit behavior of eigenfunctions, at least if the 
singular manifold is isometric to a smooth one (think of bending a sheet of paper). The case of the 
intersection is more complex, but intersections of co-dimension two or more seem to have no effect 
on the limit behavior of eigenfunctions (in L2 norm). 

6. NormaHzed Laplacians. Multiple manifolds. While we do not provide a detailed discussion 
due to space constraints, our analysis can be fairly straightforwardly generalized to other versions 
of graph Laplacians, including the random walk normalized, symmetric normalized, and twice nor- 
malized graph Laplacian discussed in Coifman and Lafon (2006); Hein et al. (2007). Interestingly, 
the popular symmetric normalized Laplacian does not lead to Neumann boundary conditions as a 
density term appears inside the directional derivative, i.e., \/ p{x)dn[f (x) / ^/ p{x)]. The resulting 
expression involves derivatives of the density and does not appear to be a natural boundary condi- 
tion. 

The analysis can also be extended to the case of multiple manifolds. We restrict our discussion 
to the case of two manifolds to simplify the exposition. 

Finally, some experimental results can be found in Appendix C. 
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2. Problem Setup 

Let Q be the set of k smooth compact Riemannian submanifolds of intrinsic dimension^ d embedded 
in M^. For each smooth component Qi, let Qi denote its interior and d^i the boundary of fij. We 
assume the boundaries dQi satisfy the necessary smoothness conditions'^. Let Q, be the union of Qi, 
and be the union of dQi. and ilj may not be disjoint, and we will consider the following two 
ways that they can associate with each other: they intersect in their interior (i.e, ilj n / 0) or 
they are "glued" along (part of) their boundaries. More precisely, given a point x G we say that 
X is regular if it falls in the interior of exactly one submanifold ilj; that is, x ^ Qi,x ^ 7^ i- 
Otherwise, x is a singular point. There are several possible singularities, and this paper considers 
the following three general classes: 

1. boundary-type: Boundary points that belong to exactly one smooth submanifold Qi'. that is, 

X G dQ,i,x ^ / i; 

2. intersection-type: Points at the intersection of two smooth submanifolds. For simplicity of 
exposition, we assume that the intersection happens between the interior of the submanifolds; that 

is, X £ Qif] / j; 

3. edge-type: Boundary points that belong to the boundaries of two smooth submanifolds. That is, 

X G d^li n d^lj,i 7^ j. 

For technical reasons, we will assume that the set of singular points is also a smooth manifold 
of lower dimension, at least locally. Moreover, for edge and intersection singularities the tangent 
space at each point of the singular manifold is the intersection of the tangent spaces to each "piece". 
We will call a union of smooth manifolds $7 with the above types of singularities a singular man- 
ifold. Notice that for a regular point x, the manifold is smooth at x, and the tangent space at x is 
homeomoiphic to M'^. 

For a singular manifold Q, we consider the following piecewise-smooth function / : 1— )■ M. In 
particular, set fi := /|q , « = 1, . . . , /c, be / restricted to the submanifold rjj. We require that each 
fi is C^-continuous on interior points, let p{x) be a piecewise smooth probability density function 
on and we assume that < a < pi{x) < b < 00. 

Graph Laplacian. Given n random samples X = {Xi, • • • , X„} drawn i.i.d. from a distribution 
with density p{x) on 17, we can build a weighted graph G{V, E) by mapping each sample point Xi 
to vertex Vi and assigning a weight Wij to edge Cjj. One typical weight function is the Gaussian, 
which is used in this paper and defined as follows: ^ 

Notice that in this Gaussian weight function, the Euclidean distance is used. The normalization by 
is for convenience of limit analysis. 
Let Wn,t be the edge weight matrix of graph G with Wn,t{hj) = Wn,t{Xi, Xj), and Dn,t be a 
diagonal matrix such that Dn,t{i, i) = X]j Wn,t{Xi, Xj), then the unnormalized graph Laplacian is 
defined as the n x n matrix L„ ^ = L>„ ^ — Wn,t- Now for a fixed smooth function f{x), and any 
point X £ Q, the graph Laplacian applied to this function / is thus 

1 " 

Ln,tf{x) = - Y,Kt{x,X,)[f{x) - f{Xj)]. 



3. For convenience we assume that the submanifolds have the same dimension. However, the same analysis can be 
applied to a mixture of manifolds with different dimensionality after normalizing the probability density. 

4. We require that the derivatives of the first two orders of the transition map exist, i.e., is a C^-manifold 
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Limit behavior of graph Laplacian. The limit study of graph Laplacians primarily involves the 
limits of two parameters, sample size n and weight function bandwidth t. As n increases, one typ- 
ically decreases t to let the graph Laplacian capture progressively a finer local structure. With a 
proper rate as a function of n and t, the limit of L^^t and its various aspects at regular points, in- 
cluding the finite sample analysis, are studied in Belkin (2003); Lafon (2004); Hein (2005); Singer 
(2006); Gine and Koltchinskii (2006); Hein et al. (2007); Belkin and Niyogi (2008a,b). The basic 
result is that the limit of L„_t/(x) at a regular point x is Ln,tf{x) A — ^7r'^/^p(x)Ap2/(x) = 
— ^7r'^/^p(x){^div|j»^grad f(x)]}, where Ap2 is called the weighted Laplacian, see Grigor'yan 
(2006). See Hein et al. (2007) for the Umit analysis of other versions of graph Laplacians at regular 
points. 

3. Limit Analysis of Graph Laplacian on Singular Manifolds 

The limit analysis of graph Laplacians typically involves limits of two parameters, the sample size 
n and the (Gaussian) kernel bandwidth t. This is usually done in two steps, first analyzing the 
limit t — )• for infinite data and then obtaining finite sample results and rates as n — )• oo using 
concentration inequalities. 

In this section we analyze the behavior of the infinite graph Laplacian (i.e., Ln,t,n = oo), 
Ltf{x), when x is on or near a singular point, t is small and the function / : — M, is fixed. Finite 
sample results and rates are given in Section 4. 

For a fixed t we define Lt as the limit of L„ ^ as the amount of data tends to infinity: 

Ltf{x) = L^,tf{x) = Ep(x) [Ln,tf{x)]f{x) = - [_Kt{x, y){f{x) - f{y))p{y)dy. (1) 

Local coordinate system: The above integral is defined on the manifold Q. In order to study the 
behavior of this integral for a small t, we need to introduce a local coordinate systems at x. The most 
convenient coordinate system for our purposes'* is obtained by a local projection from the manifold 
to its tangent space tTx : ^ ^ T^. This projection is one-to-one and smooth for points sufficiently 
close to X. Hence its inverse, vr^^ exists as well. Note that on an intersection or edge singularity the 
projection is onto the union of tangent spaces. However, it is still one-to-one as long as we restrict 
the projection of points from each piece to their respective tangent space. 

In what follows, we often use change of variables to convert an integral over a manifold to an 
integral over the tangent space at a specific point. The following bounds are used throughout the 
paper. For two points x,y £ ilj, let y' = 7r^(j/) be the projection of y in the tangent space of 
ilj at X. Let JiTxly (resp. J7:~^\y') denote the Jacobian of the map tTx at point y £ Qi (resp. of the 
inverse map vr^^ at y' € T^). For y sufficiently close to x, we have (e.g., Niyogi et al. (2008)) 

\\x - y\\ = \\x -y'\\ + Oi\\x - y'f) \\x - yf = \\x - y'f + 0{\\x - y'f), (2) 

I Jtt^Ij, - 1| = 0{\\x - yf) and | Jvr-\, - 1| = 0{\\x - y'f). (3) 
3.1. Boundary-type: Manifold with Boundary 

We first analyze the limit of Ltf{x) when x is near a smooth boundary of a single smooth manifold 
as the results for the boundary case are simpler to state. 

5. We note that another possibihty is to use the coordinate system given by the exponential map (as in Belkin (2003); 
Coifman and Lafon (2006)). This has an advantage of being independent of the embedding, but requires more subtle 
analysis in the presence of singularities. 
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Theorem 1 Let f £ p{x) £ C°°{Q) with < a < p{x) < b < oo, and d^} be the smooth 

boundary of 0,. Given a point x near the boundary, let xq be its nearest neighbor on the boundary 
dQ, and n the inward normal direction ofU, at xq. Put \\x — Xo|| = r^/t. For t sufficiently small we 
have 1 ^{d-i)/2 / 1 \ 

Ltf{x) = -_^_e-^ p(xo)5„/(xo) + o j . (4) 

Consequently, if x is on the boundary, i.e., r = 0, then x = xq and 

1 ^(°'-l)/2 / 1 

Ltf{x) = p{x)d„f{x) + 

The proof of the theorem can be found in Appendix A. 1 . For comparison, the corresponding 
result for interior points is as follows: 

Ltfix) = -i7r'^/2p(x)Ap2/(x) + 0(1), (5) 

Hence the graph Laplacian has a different behavior on or near a boundary point from interior points. 
The values of Ltf{x) are of different order, 0{l/y/i) near the boundaiy as opposed to 0(1) in the 
interior. Intuitively, the scaling difference stems from integrating over a half-plane for boundary 
points versus integrating over a (high-dimensional) plane for points away from boundary, which 
causes the integration of the first-order derivative terms non-vanishing in the former case. 

In practice, we do not know the boundary, and apply the same global normalization for all 
x £ Q. The result is that the large values of Ltf{x) are likely to correspond to points near the 
boundary (or other singular set, see below). 

We think that these observation could lead to useful techniques for data analysis and algorithms 
that respect singularities. See Appendix C.2 for some experiments on MNIST dataset. 

3.2. Intersection-type: Intersection of Manifolds 

We now present results on the behavior of the graph Laplacian near the intersection of two d- 
manifolds ili and VL2 embedded in with N > d. Note that we do not assume that the intersection 
is of codimension one. We also remark that while the boundary behavior is caused by the integration 
over a half-disk (which causes asymmetry in the integration of first-order terms) for boundary points, 
this is not the case for points around intersection singularity and edge-type singularity. The behavior 
around these latter two types of singularities are more involved and are more subtle to analyze. 

Theorem 2 Let Vli and VI2 be two d-dimensional smooth manifolds in potentially with bound- 
aries, and their intersection f^i n Vt2 is a smooth manifold of dimension l{< d — 1). Let f be 
a continuous function over $7 = Qi U whose restriction fi := fl^, on i = 1,2, is C"^- 
continuous. Given a point x £ Vti near the intersection, let xq be its nearest neighbor in R O2, 
and xi (resp. X2) be its projection in the tangent space ofVti at xq (resp. in the tangent space of 0.2 
at Xq). Put \\x — xqW = r\ft. For a sufficiently small t, we have 

Ltfix) = i=^'^/Ve-^''^-'^p(xo)(a„Ji(xo) + cosed„j2{xo)) + o (^-^^ , (6) 

where ni and 112 are the unit vectors in the direction of xq — xi and xq — X2, respectively, and 6 is 
the angle between ni and n2. 
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Proof. Given x € Qi and its nearest neighbor xq in Ui n recall that 
xi and X2 are the orthogonal projection of x onto the tangent space 
of 0,1 at xo, and onto the tangent space T^o,f22 of ^2 at xq, respectively. For 
any y G 0-2, let y' denote its orthogonal projection onto the tangent space 
7x0,02- right figure for an illustration. By the definition of Lt, we 

have Ltfix) = \ J^Kt{x,y){f{x) - f{y))p{y)dy. Since H = Hi U m, 
this can then be decomposed as 

Ltf{x) = -I' Kt{x,y){h{x) - h{y))p{y)dy + \ I Kt{x,y){h{x) - f2{y))p{y)dy (7) 



The first term above, which is the integral over ili, is exactly the graph Laplacian of /i at an 
interior point x of Oi. Thus, it is bounded by 0(1). We now focus on the second integral over i^2- 
Let M{x) denote the ball in center around x with radius t2~^ for a sufficiendy small constant 
e > 0. In the derivation below, we will approximate the integral over the manifold 1^2 by an integral 
over the region M{x) (10.2 ^ O2. The error term induced is 0(e~* = o(l) for small t. Using the 
fact that /i(xo) = f2{xo), we have: 

J [_ Kt{x,y){h{x) - f2{y))p{y)dy = -! _ Kt{x,y){h{x) - f2{y))p{y)dy + o(l) 
^ Jn2 ^ JV,{x)r\Q.2 

If If 

Kt{x,y){h{x) - fi{xo))p{y)dy + - / Kt{x,y){f2{xo) - f2{y))p{y)dy + o(l) 



=hf{x)-f{xo)) [ _ Kt{x,y)p{y)dy + - [ _ Kt{x,y){f{xo) - f{y))p{y)dy + o{l). 

(8) 

Since ||x — xo|| = r^/t, it follows from Eqn (2) that ||xo — xi || = r^/t + 0(t^/^). It can then be 
shown that ||x — X2II = r\/t-sm.6 0{t) and ||xo — X2\\ = r\ft ■ \ cos 6*1 + 0{t). For simplicity, let 
0{t^) denote 0{t^~'^) for a sufficiently small positive constant e. For a point y e ]B(x) n O2 and its 
projection in Tajp^Qj, we have that ||x — y|| = 0(^2 ) and ||xo — y|| < ||xo — x|| + ||x — y|| = 0(^5). 
This implies that \\y — y'\\ = 0{t) and ||x — = 0(t2 ) by Eqn (2). Using these distance bounds, 
we have: 

= ^e-ll-^'ll^/* . (1 + d{t)) . (1 + 0(tV2)) ^ _Le-ll-.'llV* . (1 + o(tV2)) 



^e-^-^ • (1 + 0(tV^)) = ^,{e-'-'^ ■ eOi^%-^ • (1 + 0(tV^)) 
1 g-r2sm2eg-|ix2-s/'f/t^^^Q^^i/2)) ^ e-*-' ^i^t (x2 , y') (1 + • (9) 



Now consider the first term in Eqn (8). Let tt^q denote the projection map from O2 onto Tao,f^2 
andvr"^ its inverse. Denote by B := 71^.3 (]B(x)nTa;p^n2) to be the projection of B(x)nTa:(, f22 on the 
tangent space T^.;, By applying Taylor expansion to /(x) at xq and p{y) at xq, and combining 
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Eqn 2 and 3, we have that: 

^{f{x)-f{xo)) [ Kt{x,y)p{y)dy = i((V/(xo), xi - xq) + 0(t)) / Kt{x,y)p{y)dy 



* 7B(x)nr22 * ^B(a::)nr22 

:4r(^-(V/(xo),ni)+0(ti)) 



(e-'- '^e (1 + O(t2)))(p(xo) + O(t2)) 



d+l 

t~ Jb 



\{rd„Jiixo) + 0(tt))e-'-^^'"^^l + O(tt))(p(xo) + 0(t^))( / e'^^dy' + 0(1)) 



t 2 •'TxQ,n2 
=l.C3p{xo)re-'"''-"'d^Ji{xo) + o (i=) , (10) 

where C3 = Jj, ^ e"l!''l'^dn = /jg^ e^H^H^du = tt"'/^. Note that we use the fact that -^d{\^) = 
O(^) = o (^-^^ in the above derivation. From the first line to the second line we perform a change 

of variable. From the third line to the fourth line in the above derivation, we relax /ge-ll^^^^^'ll'/^dy' 
tobe/^^ ^ e-ll^2-y'|lVtdy' + 0(e-*"') = /^^ ^ e-ll^2-y'llVt(iy'+0(l) based on the result from 
Appendix B in Coifman and Lafon (2006). 
For the second item in Eqn (8), we have 



Kt{x,y){f{xo) - f{y))p{y)dy 

(x)nn2 

Kt{x, y){f{xo) - f{x2) + f{x2) - f{y))p{y)dy 

B(x)nn2 



=kf{xo)-f{x2)][ _ Kt{x,y)dy + j [ _ Kt{x,y){f{x2) - f{y))dy 

~C3p{xo)rcos0- e-'"''''''d„j2{xo) + o (11) 

Intuitively, the first term in the third line is bounded by the quantity in the last line by a similar 
argument as the one carried out in Eqn (10) above. For the second term in the third line, observe 
that if we replace the kernel Kt{x,y) by Kt{x2,y), then j JM{x)nn2 

Ktix2,y)if{x2) - fiy))dy 

roughly corresponds to the standard weighted graph Laplacian of function /2 at an interior point X2 
in i^2- Furthermore, by a derivation similarly to the one in Eqn (9), replacing Kt{x, y) by Kt{x2,y) 
only changes the integral by a factor of e"*" ^. Hence the second term can be bounded by o(-^). 

See Appendix A.2 for details ^. The theorem then follows from Eqn 10 and 11. ■ 
From the theorem, we can see that for a point x on the intersection or near intersection, Ltf{x) 
is of the form -^Cre"'' where the coefficient C is determined by the derivatives of / and the 
position of x. Furthermore, for a point x on the intersection, we have r = 0. Hence it follows that 
the order of Ltf{x) at an intersection point is the same as those at regular points, i.e., 0(1), instead 
of order 0(1/ ^/i) as for points near singularities. 



6. Note X2 is in TxQ,n2 ■ In Eqn (1 1), to illustrate the intuition behind the derivation, we abuse the notation slightly and 
use f{x2) to refer to f{-K~^{x2)), where 7r^^(x2) is the point from 0,2 whose projection in TxQ.ii2 is X2- 
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3.3. Edge-type: Gluing Boundaries 

We now consider the case where two manifolds are glued along (a part of) their boundaries. The 
behavior of graph Laplacian for edge-type points is in some sense the combination of that for 
boundary-type and for intersection-type points: locally, a point can both "see" the boundary (thus 
will have terms arised from boundary effect) and the other manifold (which has effects similar to 
those produced by Eqn (8)). See Appendix A.3 for the proof of the following theorem. 

Theorem 3 Let (i = 1,2) be two d-dimensional smooth manifolds in with interior and 
nonempty boundary dO,i. Assume their shared boundary dili D 80,2 is a d — 1-dimensional smooth 
manifold. Let f be a continuous function over Q = 0,i U 0,2 whose restriction fi := /|q , i = 1, 2 
is -continuous. Given a point x G Jli near the glued boundary, let xq be its nearest neighbor in 
dQi n 802, and put \\x — xq\\ = r\ft. Then for a sufficiently small t, we have that 

Ltf{x) = --Lp(^o)[a(r, ^)a„ J(xo) + /3(r, 0)a„ J(xo)] + a , (12) 

where 6 is the angle between the two inward boundary normal ni and 112 of 8O1 and 80,2 at xq 
respectively, 

and <I>(x) is the cumulative distribution function of the standard normal distribution. 
Furthermore, ifx is on the edge, i.e., r = 0, then we have 

Ltfix) = -±=pix)n^'i-^y'[d„Jix) + + a (i= 

The theorem indicates that for a point x near 8O1 n 8O2, the limit of Ltf{x) is close to the 
weighted sum of two normal gradients. As x moves to the set 8O1 n 802, the weights for the two 
normal gradients will be equal. Finally, for a point x on the boundary 8O1 n 8O2, the behavior 
Ltf{x) is equivalent to the addition of the two boundary effects from Oi and O2 together. 

Intersection-type singularity again. Notice that for the case of two manifolds Oi and O2 inter- 
sect at r^i n of co-dimension 1, we can actually regard this scenario as four pieces of manifolds, 
O'f , 0~[ , O2 and O2, glued together by Oi n O2. Here, we use Of and 0^ to denote the two pieces 
of Oi but on the different side of Oi n O2. The advantage of taking this view is that we allow the 
functions on manifolds Oi and O2 to be only C'^-continuous at the intersection Oi n O2, instead of 
C^-continuous as required in Theorem 2. (However, note that Theorem 2 also holds for the cases 
when the co-dimension of the intersection is higher than 1.) Using the same argument as in the 
proof of Theorem 3 to this case of gluing 4 d-manifolds along a common d — 1-boundary, we obtain 
the following corollary. 

Corollary 4 Let Oi and O2 be two d-dimensional smooth manifolds in potentially with bound- 
aries, and their intersection ili H is a smooth manifold of dimension 1. Let f be a continuous 
function over $7 = l^i U whose restriction fi := on Oi, i = 1,2, is -continuous at any 
regular point and -continuous at points in Oi n O2. Given a point x £ Oi near the intersection, 
let Xq be its nearest neighbor in Oi O2, and xi (resp. X2) be its projection in the tangent space of 
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Jli at xq (resp. in the tangent space of 0.2 at xq). For a sufficiently small t, we have 

TT^'^-^')/^ 1 , _ , _ / 1 \ 

Ltfix) = -^p(x)[(9„i/f (xo)+9(_n2)/i (xo)+<9n2/2 (a;o)+9(_„2)/2 {xo)] + l-j= \ 

(13) 

where 9ni/]^(xo), <9(_„^)/j~(xo) are the directional derivatives of fi on two sides ofQi n ^2, same 
for dni ft {xo), f2 {xo)- 

Interestingly, it appears the eigenfunctions for a singular manifold with intersection-type sin- 
gularity can be only C'^-continuous across the intersection (instead of C^-continuous) within each 
piece of manifold. Hence the above Corollary will be useful in analysing the eigenfunctions around 
intersection singularities (or co-dimension 1). 



4. Finite Sample Complexity and Convergence Rate 

Theorem 5 Let xi, • • • , Xn, be i.i.d. random variables in sampled from a probability distribu- 
tion on ^} with the intrinsic dimension d with density p{x), < a < p{x) < b < 00, defined on the 
manifold on with the intrinsic dimension d. Let f be a bounded function, < M. Then for 

any x ^ Q.we have 

P (|L„,/(.) - L,/(.)| > .) < 2„exp (^-^^,_^^ j (14) 

where C„ and Cm ^re constants depending on the manifold. 

The proof is based on an application of the Bernstein inequality and the union bound and can be 
found in Appendix B. 

The immediate corollary is that to get an asymptotic convergence for an arbitrary fixed point (that 

is an error of the order of o(l)), we need to select t so that nt'^l'^'^'^ j log(n) — )• 00. That is, we can 

2 

choose t = (log(n) /n) ''+'1 g{n), where g{n) is an arbitrary function such that lim„__>oo g{n) = 00. 

On the other hand, for points near the singular set the scaling of the operator L„ ^ changes. 
While inside the domain Ln,tf{x) = 0(1) for a smooth (fixed) function /, for x on the sin- 
gular set or sufficiently close to it (within y/t distance), Ln,tf{x) = 0{-^). Thus an accurate 

estimate for the appropriately rescaled operator requires that nf^/^^^ / log{n) — )• 00 leading to 

2 

t = (log(n)/n)<*+2 5r(n). This may seem counter-intuitive as fewer points are required on the 
singularity than inside the domain to obtain an accurate estimate. One explanation is that near a 
singularity we are effectively estimating a degree one differential operator, while inside the domain 
we are estimating the Laplace-Beltrami operator, which is degree two. 



5. Discussion of Impact on Laplacian Eigenfunctions 

Eigenfunctions of graph Laplacians obtained from data play an important role in a variety of appli- 
cations from spectral clustering to dimensionality reduction and semi-supervised learning. While 
full proof of their convergence is likely to be very subtle and is beyond the scope of this paper 
(see Belkin and Niyogi (2008b) for the proof of eigenfunction convergence for the smooth case), 
we would like to discuss the implications of our results for eigenfunctions under the assumption 
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that such convergence takes place. Let (pt be an eigenfunction of Lt (normalized to norm 1 in L2). 
We put A = limt^o Xt and (p = limt^o "^t (with the functions converging in L2 norm). We will also 
assume that the necessary derivatives exist. 

Below, we briefly discuss the impact of each type of singularity. Some numerical experiments 
are provided in Appendix C.3. For simplicity, we will assume the samples to be infinite and the 
multiplicity of each eigenfunction to be one. 

Boundary singularity: For a point x on or near (within ^/i of) the boundary our results indicate 

Xt<pt{x) = Lt<pt{x) = C^dn(t>{x) + o 

where C is a constant independent of t. Thus, for small values of t, and Xt(j)t{x) = 0{-^)d„(f>(^x), 

which is clearly impossible unless d„(pt{x) = 0{\/t). Passing to the hmit' we see that for x on the 
boundary, 3n0(x) = 0, meaning that the limit eigenfunctions of the graph Laplacian automatically 
satisfy the Neumann boundary conditions. The experimental results in Appendix C.3 are consistent 
with this finding. 

Edge singularity: Consider now a point x at the edge where two "sheets" Qi and Q.2 come to- 
gether. From Theorem 3 we have 

C 

Lt(pt{x) = -jJ,da^(t)t{x) + da^(j)t{x)] + O 

where ni and n2 are the two directions normal to the singular set at x. By an argument following 
the one above, we have dn^(pt{x) + dn2Vt{x) = 0{\/t). and hence 

da^f{x) + dn2^{x) = 

In other words the directional derivatives for the limit eigenfunction must cancel each other. This is 
quite surprising for the following reason: suppose, that the edge singularity is obtained by folding 
a smooth manifold (imagine folding a sheet of paper, creating an edge). Then the above condition 
means that the eigenfunctions of the original surface are invariant under this "folding", despite the 
fact that the graph Laplacian operator is very different near the singular set! In other words if a 
singular surface is isometric to a smooth manifolds, the spectral structure appears to be preserved, 
even though the distances computed for points near the edge are very far from the intrinsic distances 
on smooth and the operator near the edge is not the Laplace-Beltrami operator. 

Some numerical illustrations of this phenomenon are given in the Appendix C.3. 

Intersection singularity: For the intersection singularity the analysis follows that of the edge 
case, but the implications for the shape of eigenfunctions are not completely clear, and warrant 
further investigation. One intriguing observation is that if the co-dimension of the singularity is 
greater than one, it has only a local effect on the shape of eigenfunctions. The situation is analogous 
to the disjoint union of manifolds. The reason is easy to see - the volume of \/t-neighborhood B 
around the singular set of co-dimension at least two, will be at most 0((\/t)^) = 0{t). Recall that 
an eigenfunction minimizes the quantity {Lt<j)t,4>t) L2(p) under orthogonality conditions. We see 
that the contribution of the points around the singular set is bounded by 0{t)0{^) = 0{y/t) and 
vanishes as t — 0. 

7. Since this is an informal argument, the subtleties related to convergence in L2 are ignored. 
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Appendix A. Proofs for Theorems on Graph Laplacian Limit 
A.l. Sketch of Proof for Theorem 1 

For a sufficiently small t, let il^d be the set of points that are within distance t^/'^~^ from the 
boundary dO, (a thin layer of "shell"), where e is any sufficiently small positive constant. Set 
Qin = ^/Qbd- We will show that for a small t, Ltf{x) is approximated by two different terms 
on ^}(ib and Qin, and more importantly they have different orders of t. 

Points away from boundary. Given a point x € il, let M{x) denote the ball of radius t^/^"*^ 
centered at x, where e is any sufficiently small positive constant. First, we approximate the inte- 
gral in Eqn (1) when the integral is taken inside the ball B(x). By results from Appendix B of 
Coifman and Lafon (2006), the error induced by constraining the integral to within ]B(x) n O is only 
0{e-'~'). 

Now given a point y G M{x), let u be the projection of y onto the tangent space T^^n of ^ at 
X. Choose X to be the origin of Tx^n- We have the following approximations for each of the three 
terms in the integral in Eqn (1) (see e.g, (BeMn, 2003, Chapter 4.2) and (Coifman and Lafon, 2006, 
Appendix B)). Here K{a) denotes K{a) = e~~ . N and d are the dimension of the ambient space 
and of the tangent space, respectively. 

K{\\x-y\\l^)= i^(||n||2,) + 0(||^||4^) 

f{x)-f{y)= -u^Vf{x)-\u^H{x)u + 0{\\u\\l,) (15) 

p{y)= p{x) + u^Vp{x) + 0{\\u\\\^) 

where H{x) is the Hessian of f{x) at x. 

Now for simplicity let 0{t^) denote 0(t'^~'^) for any sufficiently small positive constant e > 0. 
Let TTa; : — Tx^n be the projection from 17 onto the tangent space T^^q., and set R := iTx{M{x)r\Q). 
Obviously, R C T^^n- Combing the above approximations together with the fact that y G B(x) and 
thus \\x — y\\ = 0{t), and using a change of variable u — ^ \^w, we obtain the following: 



Ltfix) = \j^Kt{x,y){f{x) - f{y))p{y)dy 

= i lMi.)nnKtix,y)ifix) - f{y))p{y)dy + 0(e-*~^) 

{p{x) + Viu'^Vp{x))]t'^/^J7r'^\udu + i • 0(ti/2) 

[w'^'Vf{x) X w^Vp{x) + ^p{x)'uF H {x)w]}dw + o{-^) 

(16) 

From the first line to the second line, we replace the integral over Q with ball M{x), generating an 
exponentially small error for sufficiently small t (Coifman and Lafon, 2006, Appendix B). From 
the second line to the third line, by changing the variable from y to iTx{y) = u, this integral can be 
rewritten as an integral over the region R C Tx^n- In the fourth Une, we relax this integral over R to 
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be the integral over the entire tangent space Tx^n, plus another exponentially small error 0(e * 
which is consumed by o(-^). 

For an interior point x G r^j^, T^^n is a d-dimensional linear space, and the function e a'* is 
an even function on Tx^n- When taking the integral, the first term in the last integral in Eqn (16) with 

order O ^-^^ is odd and therefore vanishes. The remaining three terms in the last line of Eqn (16) 

are of order 0(1) inside the integral, and they are exactly the weighted Laplacian at x as previously 
studied in the hterature, and hence Ltf{x) = o{^). The kernel is however not symmetric for 
points on or near the boundary, which are studied next. 





/ t 




-z 


[ 


► 

-« X n 



Figure 1 : Gaussian weight at x near the boundary. 



Points near boundary. We now consider x G $7;,^ (the "shell") near the boundary. See Figure 1, 
where we show the local neighborhood around x projected in Tx^n- Let xq be the nearest boundary 
point on dU to x. Choose a local coordinate system of Tx^n with x at the origin, and assume 
the projection of xq in Tx^n is at —z. Let n the unit direction along x — z. When x is close to 
the boundary, the kernel Kt{x,y) is no longer symmetric. In particular, consider an orthonormal 
coordinate system around x, the Gaussian kernel is symmetric in all coordinate-axis other than along 
X — z. Along the direction n, the Gaussian convolution is from —zj^fi (as we have changed variable 

from u — )■ \ftw) to +oo, which is not symmetric. Therefore, e~""'"K'* is not an even function in the 
normal direction n, and the integral of the highest order term (i.e, the first term of O (^"^)) in 
last integral of Eqn (16) will not vanish. 

Specifically, let u = (ui, U2, ■ ■ ■ , u^) be the coordinate of u in the aforementioned coordinate 
system of Tx^n- Assume that ui is the coordinate along the normal direction n. Recall that we 
have applied the change of variable u — )• \/tw. So let w = {wi,W2, • • • , Wd) be the coordinate of 
w. Since we have assumed in the theorem that ||x — xo|| = r\/i, we have that = + 0{r'^). 
When we integrate the last integral in Eqn (16), all the odd terms of Uj still vanish in all directions 
except the normal direction n, and the most important point is that the leading term along the normal 



16 



Graph Laplacians on Singular Manifolds 



direction is of order O (^-^^ ^ different from 0(1) for the interior points. In particular, we have: 



-j^ /■+00 r+oo roo ^ / ^ 

Ltf{x) = -=p{x)dnf{x) I ■■■ / e "'"^^it''widwidw2---dwd + o[—= 

yt J-oo j~oo J-z/Vi \vt 

= -^-2-^e 9„/(x) + o(-^). 
The theorem follows from further performing the Taylor expansion of /(x) at xq. 
A.2. Details for the Intersection Case 

Our goal is to bound the integral j fs,(^)^n^ Kt{x,y){f{xo) - f{y))p{y)dy as in Eqn (11). First, 
recall that tTxq : ^2 '^'xo,n2 "^^e projection map from to the tangent space of Q2 at xq. 
For points on Q2 sufficiently close to xq, this map is in fact a bijection and its inverse vr"^ exsits. 
Hence for t sufficiently small (and thus X2 is close enough to x), vr^^(x2) exists and we set it to be 
X3 := vr"-^ (X2) G ^2- Since ||xo — X2 1| = r cos OVt + 0{t), we have ||x2 — X3 1| = 0(t) by Eqn 
(2). It follows from triangle inequality that ||x — X3II = ||x — X2II + 0{t) = rsm9\/t + 0{t). We 
can then apply a similar derivation as in Eqn (9) to show that 

Kt(x,y) = e-^'^'-''Kt(x3,y)-{l + d(Vi)); and Kt{x,y) = e-^' ' Kt{xs,y') ■ {l + d{Vi)), 

(17) 

where y' is the projection of y on T'xo,rj2 as defined in Section 3.2. We now have 

7/ _ Kt{x,y){f{xo)-f{y))p{y)dy 

^ (/(xo) - /(xs)) / _ Kt{x, y)p{y)dy + - I _ Kt{x, y){f{x^) - f{y))p{y)dy 

(18) 



t 



Following the same derivation as in Eqn (10), and combining with ||xo — X2\\ = r cos 9y/i + 0{t), 
the first term can be bound by: 



]:if{xo)-f{x3))l _ Kt{x,y)p{y)dy 
{x)nn2 



t 



=^(lko - X2\\d„j2{xo) + 0(t^))e-^''^'"'^(l + O(ti))(p(xo) + ©(d)) / Kt{x2,y')dy' 

= -^C3p(xo)rcose-e-'-'^i-'^9„j2(xo) +o(-^). (19) 

By using Eqn (17), the second term in Eqn (18) is: 
1 



Kt{x,y){f{x3) - f{y))p{y)dy 

(x)nn2 



e-''^"^''{l + d{Vt)) ■ Kt{xs,y){f{xs) - f{y))p{y)dy 



9 - 9 

_g— r sin 



(x)r\n^ 

I + d{Vt)) _ Kt{x3, y){f{x^) - f{y))p{y)dy + o{\). (20) 
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1 



Now observe that since the radius of M{x) is @{t'2 for a sufficiently small positive £, and since 
the distance from x to X3 is 0(t^/^), we have that there exists some ball M'{x3) of radius still in the 
asymptotic order of Q{t2~^) around X3 such that ]B'(x3) is contained inside M{x). This implies that 
M'{x3) (1^2 B(x) n ^2- Hence by results from Appendix B of Coifman and Lafon (2006), 

_ Kt{x3,y){fix3)-fiy))p{y)dy= [ _ Kt{x3,y){f{x3)-f{y))p{y)dy+0{e-'-'). 

On the other hand, } f^,^^^)r!a2 ^t{x3,y){f2{x3) - f2{y))p{y)dy h simply an approximation of the 
functional Laplacian Ltf2{x3) of the function /2 at an interior point X3 over a manifold 0,2- Hence 
by Eqn (5), 

7 / _ Kt{x3,y){f{x3)-f{y))p{y)dy = -\7T''/M^3)Ap2f2{x3) + o{l). 
It then follows from Eqn (20) that 

Kt{x, y){f{x3) - f{y))p{y)dy 



-I 



x)nn 



If , _ . „ 1 



=e-''""''7 / _ Kt(x3,y)(/(x3)-/(y))p(y)dy + o(-=) 

= - ^vr<^/2p(x3)Ap2/2(x3) + o{l=) = o(i=). (21) 
Putting Eqn (18), (19) and (21) together, we conclude that 

i / _ i^t(x,y)(/(xo)-/(y))p(y)dy = 4=C^3P(xo)rcose-e-'^'^-'^5„j2(:co) + o(^), 
which is what we claimed in Eqn (11). 
A.3. Proof for Edge-type Points 

Suppose we have two manifolds forming the "edge", denote them by f^i and $72- We use the same 
notation as in the proof for the intersection case; see the illustration in Figure 2. In particular, let xq 
denote the closest point from Oili n 80.2 to a point x G ili. Let xi and X2 be the projection of x on 
the tangent space T^g^n-^ and T^^^n^ at xq, respectively. Note that \\x — xo|| = r\^. Let /i, /2 be 
the restriction of / on ili and Q2 respectively; we know that /i(xo) = f2{xo)- Again, M{x) denotes 
the ball centered at x with radius 1 2~^. Then we have 

^ ^ Kt{x, y){f{x) - f{y))p{y)dy = \ [ _ Kt{x, y){f{x) - f{y))p{y)d + o(l); 

[x)r\Q. 



tJn t 
and 

^1 _Kt{x,y){f{x) - f{y))p{y)dy 
I JM(x)nn 



{x)nn 

If If 

Kt{x,y){fi{x) - h{y))p{y)dy + - / _ Kt{x,y){fi{x) - f2{y))p{y)dy 
B(x)nni JB(x)nC2 



t 



(22) 
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Figure 2: Edge case. 



The first integral in Eqn (22) on B(a;)nr2i is exactly what we had before for the boundary singularity 
(recall the second line in Eqn (16)), which we have shown to be the following in Appendix A.l: 

- ^ • ^vr('^-i)/V(^o)e-'^'(V/i,ni) + o (i=) . (23) 



Now consider the second integral in Eqn (22) on B(a;) n 
Kt{x,y){fi{x) - f2{y))p{y)dy 



Kt{x,y){fi{x) - f2{xo) + f2{xo) - f2{y))p{y)dy 



1 
1 

4 / _ Kt{x,y){fi{x) - fi{xo))p{y)dy + ]■ [ _ Kt{x, y){f 2{xo) - f2{y))p{y)dy 

^{fi{x)-fi{xo))l Kt{x,y)p{y)dy + \ I Kt{x, y){f 2{xo) - f2{y))p{y)dy 



t JB(a;)nf72 * JB{x)nn2 



(24) 



Let TTx denote the projection from O2 to T^jp set y' = T^xiy) for any point y G 1^2- Since 

the notation is the same with intersection case, we have by Eqn (9) 

Set B := TTx{M{x) n Q2) as the image of 18(2;) n 1^2 in the tangent space T^o.rjj- Let TJ^ 
denote the half-space of the tangent space Tx(,^n2 that contains R. Combing the bounds in Eqn (2) 
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and (3), for the first integral in Eqn (24), we have 

^ ifii^) - fi{xo)) [ Kt{x,y)p{y)dy = ];{fi{x) - fi{xo)) I Kt{x,y)p{y)J-K^^\ytdy 



i-(/i(x)-/i(xo))e-'-'^^°'^ / e-ll-^-2''ll'/*(l + 0(tV2))(p(a;o) + o(ii/2))rfy' 



t2^^ JB 

=C(r,0)i=p2(xo)re-'-'^-'^(V/i(xo),ni) + o(i=) , (25) 

where C{r,0) = e-H-^-s/'lPArfy' = J „,>_,cos(e) e-l'^H'^y = ^^/2«>(^/2r cos^), and 

is the c?-th entry of coordinate of y in the d-dimensional tangent space. The last step above also 
involves applying the following formula: 

fi{x) - fi{xo) = (V/i(xo),xi - :eo) + d{\\x - xof) = rVt(V/i(xo),ni) + 0(i). 

Now for the second integral in Eqn (24), we have: 

Kt{x,y){f2{xo) - f2{y))p{y)dy 



Kt{x,y){Vf2\,.xo - y)p{y)dy 



=7/ Kt{x,y){Vf2\ ,XQ-X2)p{y)dy + \ I Kt{x,y){Vf2\ ,X2-y)p{y)dy 

(26) 



The second term from the above equation is exactly the same as if we were considering a point X2 
nears the boundary of U2- Hence using the results from Appendix A.l, we have that 



1 



Kt{x,y){Vf2\^,X2 - y)p{y)dy 



^ _g-r^sin^( 



i2+i Jb " 



1 



e-^'^-^-'<^{p(xo) + d{t'/^)){Vf2\. / e-ll^^-^'ll'/*(x2-y)dy) 



il+i Jb 



1 



e-''^'-'%p{xo) + d{t^/^)){Vf2\, / e-\\^^^-y'^\'/\x2-y)dy) 



t2- 

: - . i7r^e-^'a„j2(xo) + o(i=). (27) 
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For the first term from Eqn (26), we have: 

_Kt{x,y){Vf2\^^^,xo-X2)p{y)dy 
{x)nn2 



1 .g-r^.in^e / e-|l-2-.'||V*(l + 0(il/2))(V/2(xo),Xo-X2)(p(xo) + 0(tl/2)W 



Jb 



1 



-e-'" '(1 + d{t'/^)){p{xo) + diVt)) I e-ll^^-J''ll'/*(V/2(xo), xo - X2)dy' 



t2^^ JB 

l-e-^'^'-'%{xo){Vf2{xo),xo - X2) [ e-^\-^-y'\^'/'dy' 



4.-.-.„..,„„.....(/' ,-.-,...,.„.-.-,) 

=l=p{xo)C{r, 9) . r cos ^e"'"' ^^'^^ ^(V/2(xo), + o , (28) 

where C(r, 6) is defined earlier. Let x' be the projection of xi on the tangent space Tx^^n^ - The last 
step in Eqn (28) uses the fact that ||xo — X2II = r\/t • | cos 9\ + 0{t). Combining all the results in 
(23), (25), (26), (27), and (28), we prove Theorem 3. 

Appendix B. Proof for Convergence Rate 

Lemma 6 Let Xi, ■ ■ ■ ,Xn, Z be i.i.d. random variables of M.^ from density p{x) G C°°(0), 
< a < p{x) < b < 00, Kt{x, y) be a Gaussian weight function and \ f\ < M, then for any x € ^2 



P 



1 

- V Kt{x, X,)f{X,) - ¥.z[Kt{x, Z)f{Z)\ 

n ^ — ^ 



n 

i=l 



^^P^^^n-^OT^ '''' 



where Cy and Cm only depend on d and K{-, ■), /(•), p(-). 

Proof LetW^i(x) = Ktix , Xi) f (Xi) , and Yiix) = Wi{x) -ExAWi{x)]. We have Exj5^i(x)] = 0, 
and 

\Yi{x)\< \Wi{x)\ + \Ex,m{x)]\ 

= \Kt{x,X,)f{X,)\ + \f^Kt{x,y)fiy)piy)dy\ 

< ^M + MbJ^Kt{x,y)dy ^^""^ 

< ^ + MbCg = (Cm + MbCgt''/^)/t''/^ 

where Cg = f-^ Ktix,y)dy < 00, and Cm = M. For Var[yj(x)], 

Var[y,(x)] = ExAW^ix)] - {ExAW^ix)]}^ 

< J^KKx,y)f{y)p{y)dy + [Jj^Kt{x,y)f{y)p{y)dy]^ 

< ^,M%j^Kt{x,y)dy + M%''Cl 

< ^M%Cg + M%^C^ = (a + MH^Cy/^)/t^/^ 

where C„ = M'^bCg. By the Bernstein's inequality 



-^''''^l {C, + M%^C^A'^/^) + {Cm + MbCgt'^/^)e/3 ^ ^ 
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Proof of theorem 5: For x S 57, we put f{x) — f{Xi) into Lemma 6, obtaining: 

P{\Ln,tf{x)-Ltf{x)\>e) 



--P 



1 " 

- V Kt{x, Xi){f{x) - f{X,)) - Ez[Ktix, Z){f{x) - f{Z))] 

^^(d+4)/2g2 \ 



n 

i=l 



> te 



The proof of tiieorem 5 is done. 
Appendix C. Experimental results 

Below we provide some numerical results validating theoretical analysis, some experiments with 
real datasets and several numerical experiments in support of our conjectures on the eigenfunction 
behavior. 

C.l. A numerical example 

In this section, we illustrate the behavior of graph Laplacian on or near the singularity points 
through a simple numerical example. In order to explore different cases, we consider the union 
of 1 -dimensional manifolds 57 as shown in Figure 3 (a), which is a combination of three linear in- 
tervals, Oi, and ^3 in M^. We take 2500 point uniformly spaced over the singular manifold and 
choose /(xi, X2) = (xi + 0.2)^ + x"^ (restricted to the manifold). 

Shape and Scaling Behavior: The value of Ln.tf{x) over all points in ili is shown in Figure 3 
(b). We can see that Ln,tf{x) near the boundary is approximately half a Gaussian function, while 
the function near the intersection is similar to a function of the form (p{x) = axe~^^ , as predicted 
by Theorem 2. The shape of Ln^tf{x) near the comer is also what we expect from Theorem 3. 

To explore the scaling effects, in Figure 3 (c) we plot values of log \Ln,tf{x)\ against log(t) for 
different values of t. Each graph corresponds to a fixed point x chosen near a boundary, intersection 
or edge singularity. Our theoretical results predict a linear curve with slope — ^ in log-log coordi- 
nates (corresponding to l/^/t, which is consistent with our experimental results as shown in (c). 



C.2. Singularity Detection and Estimation 
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(a) Sample data from $7 (Z?) L„ ^/(x) over f^i (c) log(|L„ vs log(t) 

Figure 3: Ln,tf{x) with /(xi, X2) = (xi + 0.2)^ + Xg over In (c), x axis is log(|L„,j/(x)|) and 
y axis is log t. The three curves are for three points near each type of singularity. 





(a) Data set 



(b) Values of Ln,t/ 
on Bunny when f{x, y, z) 



y- 



Bunny Model: In the following ^ ^^Ife " Hy^f) 

example, we use the union of the 
Stanford bunny dataset (in blue) 
and a plane y = (in red), which 
intersects the bunny in the mid- 
dle as shown in panel (a). We 
have take 10000 points for both 
the bunny and the plane in the point 
cloud, and choose the function / = 

y. The values of L„ 4/ on the bunny are shown in panel (b). We can see that the values of Ln,t/ 
at the intersection are and values near the intersection go up (in yellow) and down (in blue) as 
expected from the theoretical analysis of the intersection singularity. 

MNIST Data: From the scaling behavior of the graph Laplacian near singularities, it is poten- 
tially possible to detect certain such singularities on real world data sets. We use the example of the 
MNIST digit images. The image of each digit can be thought of as a sample from a low-dimensional 
manifold embedded in the high-dimensional pixel space. We choose function f{x) to be the sum- 
mation of the pixel intensity of each image. We take the images corresponding to the top 2% of 
the highest value of Ln,tf{x) as the "near singularity/boundary" images, which are compared to 
the "average" images of each digit. The results are shown in Figure (6). We see that the images 
on the right are significantly larger (and systematically different) from average, suggesting that they 
may belong to the part of the boundary, which can be detected through the sum-of-pixel-intensity 
function. 



C.3. Laplacian Eigenfunctions on Singular Manifolds 

In this section, we will provide two numerical examples to validate our to conjectures about the 
Laplacian eigenfunctions on the singular manifolds. 

Folded Rectangle: In this example, we consider two manifolds: (1) the rectangular region 
(-0.3, 0.3) X (-0.5, 0.5) X {0}, in and (2) the "folded" rectangle Q2, obtained by transforming 
the positive y part of l^i, i.e. {{x, y, z) £ Ui : y > 0}, by applying the linear transformation given 
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(a) ni (b) rt2 

Figure 4: Rectangle and folded rectangle and 0,2- 



by the matrix 



cos(7r/4) — sin(7r/4) 
sin(7r/4) cos(7r/4) 

and keeping the rest of VLi fixed as showing in Fig. 4. We see VLi and 0.2 are intrinsically isometric, 
yet ^2 has an edge singularity. 

We choose 6000 uniformly-spaced points in both Q.i and Q.2- 

With this two data sets, we construct their graph Laplace matrices respectively, and calculate 
their eigenvalues and eigenvectors using the Gaussian kernel with t = 10^^. Figure 5 shows the 
first 16 nontrivial eigenvectors of two graph Laplace matrices. We can see that the corresponding 
eigenvectors match nearly perfectly. It can be easily verify that they also match eigenfunctions of 
the rectangle with Neumann boundary conditions. 

To give some precise numerical results we measure the relative difference between the sets of 
eigenvalues. Let and X\ be the two vectors of first k eigenvalues for Oi and respectively. We 
define the (relative) difference as 



diff. 



All 



|Ai.| 



The table below shows the difference for various values of k. We observe that the approximation 
appears quite precise with difference of the order of 0.1%, thus providing further evidence in support 
of our conjecture on the behavior of eigenfunctions. 



k 


10 


50 


100 


difffc 


0.0014 


0.0012 


0.0011 
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Figure 5: 2nd-17th eigenvectors of the graph Laplace matrices of the two manifolds Qi and 0,2 



s 


1 











(a) 



ib) 



Figure 6: An averaged image of digits "1" vs an averaged image with a large value of graph Lapla- 
cian constructed from the data set of images, applied to a function defined by the sum of 
pixel intensities. 
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